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Abstract 

At  the  United  States  Air  Force  Academy  (USAFA)  and  many  other  engineering  schools,  the  culminating  experience  prior  to 
award  of  a  degree  is  a  capstone  design  experience.  The  desired  outcomes  lor  such  a  capstone  design  experience  are  very  similar 
across  engineering  programs.  Each  program  or  discipline  has  freedom  in  how  they  achieve  these  outcomes,  so  long  as  it  is  a 
deliberate  and  traceable  approach  back  to  the  desired  outcomes.  This  freedom  allows  each  discipline  to  tailor  their  capstone 
design  experiences  to  those  appropriate  to  their  domains.  When  students  are  developed  fully  within  a  single  discipline  program 
that  also  offers  their  capstone,  the  structure  promotes  alignment  of  the  student,  instructor,  and  advisor  expectations.  However,  as 
students  are  assigned  outside  of  their  core  discipline  to  support  other  capstones,  misunderstanding  of  how  their  unique  skills 
support  the  capstone  outcomes  increases.  The  ability  to  then  compare  capstones  beyond  the  top-level  outcomes  becomes  difficult. 
This  is  the  case  for  systems  engineering  (SE)  majors  at  USAFA  where  they  are  allocated  to  other  engineering  capstones.  In  order 
to  trace  these  distributed  students* 1  capstones  back  to  a  common  set  of  outcomes,  a  framework  tor  understanding  the  full  spectrum 
of  their  experiences  is  needed.  This  paper  will  review  previous  work  in  characterizing  capstone  experiences,  present  the  method 
used  to  frame  USAFA’s  capstones,  and  show  a  proposed  a  set  of  key  characteristics  and  associated  rubrics. 
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1.  Introduction 

The  United  States  Air  Force  Academy  (USAFA)  and  many  other  engineering  schools  provide  a  culminating 
engineering  design  experience  prior  to  award  of  the  degree.  The  desired  outcomes  for  such  a  capstone  design 
experience  are  very  similar  across  engineering  and  applied  science  programs.  Each  program  or  discipline  has 
freedom  in  how  they  achieve  these  outcomes,  so  long  as  it  is  a  deliberate  approach  and  traceable  back  to  the  desired 
capstone  outcomes.  The  overall  program  outcomes  must  be  shown  to  meet  the  Accreditation  Board  for  Engineering 
and  Technology  (ABET),  and  as  a  capstone  experience,  many  times  the  capstone  outcomes  align  very  well  with  the 
program  outcomes.  This  freedom  allows  each  discipline  to  tailor  their  capstone  design  experiences  to  those 
outcomes  appropriate  to  their  domains.  Shown  below  are  the  Course  Student  Outcomes  for  a  Systems  Engineering 
(SB)  major  participating  in  a  capstone  at  USAFA. 

1 .  Understand  and  implement  rigorous  systems  engineering  practices. 

2.  Critically  analyze  and  trade-off  program  requirements  Sc  constraints  (cost,  schedule  &  performance)  to 
develop  realistic  system  design  options. 

3.  Demonstrate  independent  learning  by  researching  and  assessing  specific  issues  of  system  performance  and 
applying  them  to  individual  team  tasks. 

4.  Demonstrate  an  ability  to  work  effectively  as  a  member  of  a  Systems  Program  Office  team,  in  both  leader 
and  follower  roles,  by:  understanding  program  goals  and  objectives;  identifying  problems,  analyzing 
alternatives  and  implementing  solutions;  diligently  tracking  and  documenting  decisions  and  analytical 
results;  and  successfully  completing  program  milestones. 

When  students  are  developed  fully  within  a  single  discipline  program  that  also  offers  their  capstone,  the  structure 
promotes  the  student,  instructor,  and  advisor  expectations  (e.g.  a  mechanical  engineering  major  completing  a 
mechanical  engineering  capstone  project  within  the  Department  of  Engineering  Mechanics  supported  completely  by 
mechanical  engineering  faculty).  However,  as  students  are  assigned  outside  of  their  engineering  discipline  to  support 
other  capstones,  potential  for  misunderstanding  of  how  their  unique  disciplinary  skills  support  the  capstone 
outcomes  increases  (e.g.  a  systems  engineering  major  completing  a  space  satellite  design  capstone  project  within  the 
Department  of  Astronautical  Engineering  supported  by  a  combination  of  astronaut ical,  mechanical,  and  systems 
engineering  faculty). 

Systems  engineers  at  USAFA  are  developed  in  a  program  that  is  based  on  the  premise  that  an  SE  should  have  a 
strong  foundation  and  ability  to  operate  capably  in  a  classic  engineering  domain.  So  by  the  SE  program's  very 
design,  SE  students  are  meant  to  be  embedded  in  other  engineering  domain  projects  to  bring  their  domain- 
independent  education  and  skills  to  bear.  But,  this  program  design  comes  at  a  risk.  The  ability  to  compare  capstones 
beyond  the  top-level  student  outcomes  becomes  a  difficult  process  without  some  way  of  characterizing  the 
similarities  and  differences  among  different  capstone  projects  themselves.  This  potential  for  misalignment  is  the  case 
for  SE  majors  at  USAFA  where  they  are  allocated  to  other  existing  engineering  program  capstone  projects.  This 
challenge  directly  applies  to  the  SE  program's  distributed  construct  where  it  is  difficult  to  establish  multidisciplinary 
teams  without  an  “overarching  college-wide  structure  In  place  to  make  it  happen.” J 

Multidisciplinary  teams/projects  are  possible  through  specific  student  enrollment  in  their  major-aligned  capstone 
course  number  and  careful  advisement  of  projects  to  enable  their  respective  outcomes.  This  is  the  case  with  SE 
students  who  are  placed  on  capstones  hosted  by  other  disciplines  (e.g.  human  factors  engineering,  aeronautical 
engineering,  electrical  and  computer  engineering,  etc.).  The  SE  students  are  physically  on  the  same  project,  but 
enrolled/advised  in  a  separate  capstone  course  number  that  maintains  the  linkage  back  to  the  SE  degree  program. 

Systems  engineering  students  are  placed  outside  of  a  centralized  capstone  course  for  several  reasons  unique  to 
SB.  First,  the  application  of  SE  depends  heavily  on  having  one  or  more  application  domains  (or  disciplines)  to 
enable  the  full  value  of  SE  to  be  realized,  ft  is  the  integration,  traceability,  and  formalized  technical  communication 
that  SE  brings  to  a  project.  Without  an  application  domain,  SE  is  reduced  to  merely  doing  systems  analysis  for 
analysis'  sake.  Second,  SE  majors  are  placed  outside  of  a  central  SE  capstone  due  to  the  current  organizational 
structure  of  the  SE  program  at  USAFA,  The  few  core  SE  instructor  billets  are  insufficient  to  exclusively  support 
capstone  projects  for  the  current  SE  student  load— roughly  65-80  students  per  class  year. 
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In  order  to  support  and  trace  these  distributed  students'  capstones  back  to  a  common  systems  engineering  set  of 
outcomes,  a  framework  for  understanding  the  full  spectrum  of  their  experiences  is  needed .  Establishing  such  a 
framework  is  difficult  because  of  the  wide  variety  of  discipline-specific  projects  that  vary  based  on  many 
characteristics  (e.g.  funding  source,  team  size,  initial  definition  state  of  customer  needs,  skill  diversity  of  the  team, 
etc.)*  This  research  seeks  to  understand  the  previous  efforts  to  characterize  capstone  experiences,  present  a  method 
used  to  frame  USAFA’s  capstones,  and  show  a  proposed  a  set  of  key  characteristics  and  associated  rubrics, 

2.  Related  work 

Previous  research  within  the  topic  of  capstones  appears  to  focus  largely  on  methods  of  assessing  capstones  for 
accrediting  bodies  such  as  the  Accreditation  Board  for  Engineering  and  Technology  (ABET)2  3.  Authors  that  were 
involved  with  their  university's  ABET  visit  documented  their  approaches  to  developing  a  rigorous  engineering 
experience  that  could  assess  their  program’s  desired  program  student  outcomes. 

There  is  also  research  that  addresses  the  comparison  of  single  dichotomous  pairs  of  characteristics  for  a  capstone 
(e.g.  single  to  multi-disciplinary'  teams4  5,  small  vs.  large  groups6,  short  vs.  long  duration  projects1,  and  determined 
which  characteristic  had  merit  for  various  stakeholders. 

Frameworks  have  been  developed  for  understanding  various  parts  and  phases  of  capstones  as  well.  One  effort 
promoted  a  Quality  Function  Deployment  (QFD)  approach  to  describing  the  integrated  engineering  design  education 
through  capstones7.  Others  have  described  the  three  key  elements  (student  preparation,  project  selection,  and 
instructor  mentorship)  that  frame  a  capstone  and  must  be  addressed  before  the  capstone  should  even  be  attempted8. 
One  group  used  an  uncertainty,  complexity,  and  pace  (UCF)  model  to  characterize  management  style  and 
application  within  the  development  of  an  Israeli  fire  control  system4.  A  group  at  the  US  Military  Academy’s  system 
engineering  department  present  four  essential  elements  (real  world  problems,  using  a  total  design  process,  closing 
engineering  competency  gaps,  and  integration  of  technical  skills)  that  are  key  in  order  to  meet  student,  faculty  and 
ABET’s  expectations  for  a  meaningful  capstone  experience9.  Related  to  capstones,  technical  projects  in  general  have 
been  evaluated  to  develop  a  framework  to  understand  "hard”  vs.  "soft”  projects  through  identification  of  seven  key 
dimensions10,  Crawford  and  Pollack’s  research  was  particularly  interesting  in  that  they  presented  a  seven-dimension 
framework  for  evaluating  whether  a  project  was  "hard”  or  "soft.”  In  each  of  these  frameworks,  direct  application  to 
the  distributed  $£  capstone  construct  is  incomplete.  Where  possible,  this  previous  research  will  be  used  to  illustrate 
effective  parallel  framing  methods  and  aid  in  characteristic  rubric  generation  for  areas  already  studied. 

The  unique  aspect  of  the  current  research  project  is  that  it  seeks  to  develop  a  more  complex  and  holistic 
framework  to  understand  the  capstone  experience  across  many  disciplines.  The  need  for  this  multi-faceted  approach 
is  unique  to  “sharing  programs”  (e.g.  system  engineering)  that  must  allocate  students  Into  projects  hosted  in  separate 
programs.  Previous  research  support  a  product  vs.  skill  teaming  construct  that  allows  for  product  focus,  but  sharing 
universal  skillsets  to  enhance  a  capstone  project5.  The  ability  for  advisors  central  to  the  “sharing  program”  to 
compare  and  contrast  the  experiences  is  important  to  ensure  that  common  capstone  student  outcomes  are  being 
attained  regardless  of  domain.  The  value  of  such  a  framework,  however,  goes  beyond  just  the  central  understanding 
of  the  capstone  landscape  and  will  provide  the  entire  capstone  community  a  shared  understanding  for  discussion  and 
improvement. 

3.  Research  objective  &  approach 

The  primary'  objective  of  this  research  is  to  analyze  the  following  three  questions  regarding  the  classification  and 
correlation  of  capstone  characteristics  with  attainment  of  the  capstone  course  student  outcomes.  Ultimately,  the 
results  of  this  research  will  inform  stakeholders  of  the  capstone  experience  and  allow  a  more  seamless  understanding 
of  how  our  students  achieve  the  student  outcomes  required  of  the  capstone  experience. 
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3.  L  Research  questions 

The  following  questions  represent  three  phases  of  investigation  that  this  research  seeks  to  explore.  The  current 
paper  is  meant  to  present  the  overall  approach  and  set-up  prior  to  collects  results  in  each  of  the  three  sequential 
phases. 

1.  Using  a  2-part  questionnaire  to  narrow  and  then  classify  capstone  experiences  using  rubrics,  is  it  possible  to 
establish  a  common  framework  for  characterizing  the  full  breadth  of  capstone  experiences  at  USAFA? 

2.  Is  there  a  relationship  between  specific  capstone  characteristics  and  student  performance  (with  respect  to 
capstone  course  student  outcomes)? 

3.  How  can  the  established  framework  and  observed  relationships  be  used  to  affect  capstone  offerings  and 
placement  of  students  in  capstones  to  improve  the  student  outcomes  of  capstones? 

3.2 .  Research  design  assessment  strategy?: 

The  first  research  question  will  be  addressed  in  the  creation  of  a  framework  for  capstone  characteristics.  This 
framework  will  form  the  foundation  of  the  research  and  will  be  created  with  inputs  of  the  many  capstone  faculty 
advisors.  Initial  discussion  with  the  capstone  faculty  yielded  dozens  of  characteristics  that  capstones  exhibit  (e.g. 
large  team*  externally  funded,  project  level,  formal,  iterative,  open-ended,  highly-constrained,  step-wise, 
homogeneous  team  members,  etc).  The  lead  author  of  this  paper  is  in  a  uniquely  suited  position  as  the  “USAFA 
System  Engineering  Capstone  Coordinator?’  In  this  role,  he  has  direct  exposure  and  understanding  of  all  30+ 
capstone  projects  currently  being  administered  at  USAFA.  Many  of  the  characteristics  identified  by  faculty  advisors 
were  observed  as  contrary  to  each  other;  forming  dichotomous  pairs  that  fell  within  a  generalized  spectrum  (e.g.  the 
spectrum  of  “degree  of  constraints”  would  include  possible  characteristics  of  “open-ended”  or  “highly- 
constrained”).  From  an  initial  list  of  19  characteristic  spectrums,  the  list  will  be  narrowed  by  way  of  a  poll  to  the 
capstone  faculty  at  large.  This  subset  of  characteristics  will  then  be  further  developed  with  rubrics  to  establish  the 
spectrum  of  characteristics  between  the  extremes.  See  section  4  for  examples  of  these  selected  and  developed 
characteristic  spectrums. 

These  spectrum  rubrics  will  then  be  used  by  faculty  advisors  to  seff-assess  each  of  the  30+  current  capstone 
projects  in  order  to  classify  the  projects  against  all  of  the  chosen  aspects.  The  authors  and  a  few  others  with  broad 
knowledge  of  the  current  capstones  will  also  assess  the  current  capstones  according  to  the  developed  rubrics.  This 
classification  effort  and  its  initial  results  will  form  a  constructivist  solution  to  the  first  research  question. 

For  the  second  research  question,  performance  data  for  the  students  in  the  2015  year  group  who  are  members  of  a 
capstone  projects  will  be  used  along  with  faculty  assessments  to  determine  attainment  of  capstone  student  outcomes. 
Final  projects,  final  grades,  and  an  end-of-course  student  survey  will  be  used  to  establish  the  student  attainment  of 
the  capstone  outcomes.  Also,  a  capstone  faculty  mentor  survey  similar  to  one  given  in  previous  years  to  advisors  of 
systems  engineering-supported  capstones  wilt  be  used  for  a  faculty  performance  assessment  of  the  students.  This 
performance  data  will  then  be  compared  with  the  scores  of  the  relative  capstone  characteristics  to  determine  if 
correlation  relationships  exist.  One  significant  challenge  exists  if  only  final  project  grades  are  used  for  comparison 
due  to  the  many  factors  that  go  into  these  (e.g.  technical  solution  success,  grade  distribution  within  a  team  of  several 
separate  majors,  instructor  grading  approaches,  hosting  department  grading  approaches,  and  other  nuanced 
differences  between  projects).  Therefore,  the  final  project  grades  will  be  carefully  considered  in  concert  with  the 
student  and  faculty  survey  results  which  will  represent  more  direct  reflection  of  outcome  attainment. 

The  presence  of  correlation,  no  correlation,  or  in-conclusive  correlation  will  be  evaluated  and  discussed  for 
possible  causes  and  potential  for  exploitation  to  improve  the  capstones.  For  example,  if  the  study  concludes  that 
there  is  high  correlation  of  outcome  success  for  projects  that  are  assessed  at  the  “open-ended”  side  of  the  “degree  of 
constraints”  spectrum,  future  projects  will  be  encouraged  to  remove  as  much  early  problem  definition  as  feasible 
while  balancing  project  progress.  Or,  if  it  is  shown  that  no  correlation  exists  between  outcome  success  and  the  “size 
of  team”  characteristic,  then  capstone  mentors  can  be  advised  to  not  worry  so  much  about  adherence  to  team  size 
constraints  and  will  be  encouraged  to  enhance  their  capstones  in  other  ways.  These  two  examples  are  only 
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illustrative  of  potential  use  of  possible  conclusions  and  should  not  be  viewed  as  actual  conclusions  at  this  stage  of 
research. 

The  third  research  question  will  be  addressed  by  the  comparison  of  performance  in  capstones  data  between  the 
2015  and  2016  year  groups  where  application  of  the  capstone  characteristic  correlations  have  been  developed  and 
applied  in  the  capstone  development  for  the  2016  year  group.  Application  of  the  conclusions  from  the  first  two 
research  questions  may  take  the  form  of  information  sharing  among  the  capstone  mentors  for  their  awareness, 
adjustment  of  specific  project  characteristics  where  there  is  clear  opportunity  to  align  with  the  desired  capstone 
characteristics,  and  possibly  down-playing  emphasis  on  certain  characteristics  where  success  of  outcomes  is  neutral 
{no  correlation  established).  Application  methods  will  depend  on  what  the  conclusions  from  research  questions  one 
and  two  are  and  the  receptiveness  of  the  various  hosting  departments  to  input  for  capstone  experience  improvement 
Similar  to  data  collection  for  question  two,  the  data  will  consist  of  final  projects,  final  grades,  and  an  end-of-course 
student  survey  will  be  used  to  establish  the  student-based  attainment  of  the  capstone  student  outcomes.  Also,  a 
capstone  faculty  mentor  survey  similar  to  one  given  in  previous  years  to  advisors  of  systems  engineering-supported 
capstones  will  be  used  for  a  faculty  performance  assessment  of  the  students* 

With  the  above  research  plan  in  place,  the  next  steps  were  to  develop  the  characteristics  that  would  be 
investigated  for  this  research. 

4.  A  proposed  framework  of  capstones 

The  following  sections  introduce  the  framework  foundation  that  will  be  used  for  the  future  research.  Assessment 
of  student  outcomes,  correlations  and  conclusions  of  all  future  research  will  be  based  largely  on  the  framework  that 
is  developed  below. 

4.  L  Capstone  characteristics 

The  first  step  in  developing  a  framework  for  understanding  capstones  was  to  understand  the  various 
characteristics  that  capstone  instructors  use  to  describe  their  projects.  These  characteristics  have  several  related 
descriptions  that  are  most  often  mentioned  when  instructors  attempt  to  place  their  project  in  context  with  other 
projects  or  to  external  visitors  (e*g.  L'My  project  is  extern  ally -funded  and  is  looking  at  the  novel  generation  of  a 
system-level  solution  to  a  highly-constrained  problem*  I'm  also  promoting  design  tool  usage  as-needed  in  a  very 
informal  way,”)*  While  this  is  what  a  typical  instructor  may  say,  what  they  are  doing  is  assessing  placement  of  their 
project  on  several  characteristic  spectrums,  or  dichotomies. 

In  order  to  focus  in  on  the  characteristics  that  are  important  to  describe  and  understand  a  capstone,  the  following 
characteristics  were  gathered  from  the  lead  author's  exposure  to  all  30+  USAFA  capstone. 

a  Funding  source  (e*g.  intemal/extern  a  i/none) 

0  Degree  of  constraints  (e.g.  open-ended,  highly  constrained) 

*  Starting  point  for  requirements  refinement  (e.g,  ill-defined,  existing  requirements) 

*  Agility  of  design  process  (e*g*  step- wise,  as-needed) 

*  Diversity  of  team  member  major/ski  11  set  (e*g.  homogeneous,  multidisciplinary) 

*  Scope  of  programmatic  concern  (e.g,  project  team,  program  office) 

*  Reflection  of  DoD  development  process  {e*g.  DoD  5000,  rapid  capability,  industry  innovation) 

*  Customer  involvement  (e.g*  internal,  external) 

®  Team  size  (e.g*  number) 

*  Novelty  of  project  (e.g.  original  problem/solution,  existing  project  framework) 

*  Formality  (e*g*  relative  number  of  formal  briefs/  reports) 

*  Potential  for  publishing  work  {e*g.  expectation,  exception) 

*  System  level  (e.g*  consumer-level  product  design,  system-level  design,  system  of  systems) 

*  Course  maturity  (e.g.  new  CD/offering,  years  of  refined  offering  of  similar  capstones) 

*  Knowledge  use  (e.g.  much  new  knowledge  required,  application  of  previously  learned  material) 
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•  Other  faculty  involvement  (e.g.  single  instructor,  team  of  instructors) 

•  Team  selection  (e.g.  volunteer  filled,  directed) 

•  Competitive  (e.g.  sole-source,  intemal/externa!  USAFA  competition) 

•  Mission  linkage  (e.g.  military  need,  general  application) 

The  example  terms  provided  in  each  line  were  what  faculty  advisors  used  to  explain  their  projects.  By  pairing  the 
related  terms,  the  more  general  name  for  a  characteristic  spectrum  was  achieved  as  an  initial  step.  It  is  suspected 
that  not  all  characteristic  spectrums  will  be  of  use  in  describing  all  current  capstones.  This  is  another  way  to  say  that 
there  is  a  relative  utility  of  description  in  the  list  of  characteristics  listed  above.  For  this  reason,  a  follow-on  step  to 
this  paper  will  be  to  poll  the  capstone  advisors  for  their  preferred  subset  of  the  characteristic  spectrums  above.  This 
will  represent  the  first  part  of  the  two-part  questionnaire  referenced  in  research  question  one  in  section  3. 1 

4,2,  Dichotomies  of  characteristics 

in  preparation  for  the  second  part  of  the  two-part  questionnaire  referenced  in  research  question  one  in  section  3.1, 
rubrics  were  developed  for  each  characteristic  spectrum.  Each  spectrum  was  evaluated  for  the  two  extreme 
characteristics  a  project  could  exhibit  in  that  aspect.  With  these  extremes  established,  the  adjacent  positions  of  the 
rubric  were  populated  with  example  characteristics  appropriate  for  the  relative  location  on  five-position  rubrics. 
These  rubric  descriptions  were  developed  by  the  authors  based  on  direct  observation  of  the  spectrum  of 
characteristics  present.  The  resulting  rubrics  are  displayed  in  the  following  tables. 


Table  I  Rubric  characterizing  the  "funding  source"  spectrum 


External 

Internal 

Sourced  primarily 
external  to  Government 
funding 

Sourced  primarily 
from  a  Government 

agency 

Shared  mix  of 

sources  internal  and 

external 

Primarily  sourced 
from  USAFA  budget 

Sourced  solely  from 
the  hosting 
department's  standard 
course  O&M  budget 

Table  2.  Rubric  characterizing  the  "degree  of  constraints"  spectrum 

Open-ended 

Highly -cons  trained 

Syllabus  topics, 
schedule,  deliverables 
and  other  aspects  of  the 
project  are  all  highly 
fluid  and  largely 

studenEled  based  on 

progress  and  needs 

Syllabus  topics, 
schedu  1  e ,  del  i  ve  tab  les 
and  other  aspects  of  the 
project  are  provided  in 
a  rough  framework 

Syllabus  topics, 
schedule,  deliverables 
and  other  aspects  of  the 
project  are  defined,  but 
regular  updates  are 
made  by  faculty  and 

students  as  needed 

Syllabus  topics, 
schedule,  deliverables 
and  other  aspects  of  the 
project  are  all  highly 
defined  and  but  can  be 

tailored  for  certain 

cases 

Syllabus  topics, 
schedule,  deliverables 
and  other  aspects  of  the 
project  are  all  highly 
defined  and  very  rarely 
deviated  from  in  the 

course  of  the  capstone 

Table  3.  Rubric  characterizing  the  "starting  point  for  requirements1'  spectrum 

ill-defined 

Existing  requirements 

Problem  statement 

known 

Initial  customer  needs 

known 

Customer  needs  and 
requirements  already 
known 

Requirements, 
interfaces  and  some 

components  already 

established 

Requirements, 
interfaces,  and  entire 
subsystems  already 

established  with  several 

other  constraints. 

System  modification  to 
a  well -dell ned/highly 
constrained  system 
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1  able  4,  Rubric  characterizing  the  "agility  of  design  process'1  spectrum 


cuep-wise 

As  -needed 

Syllabus  of  design 
topics  established  and 
followed  rigorously 

Syllabus  of  design 
topics  established,  but 
can  be  adjusted  in 

certain  cases 

Syllabus  of  design 
topics/tools  is  mostly 
defined,  but  adapted 
regularly  to  the  project 
progression 

Basic  design 
topics/tools  are 
presented  and  then 
augmented  as-needed 

Design  topics/tools 
introduced  only  as 
needed 

Table  5.  Rubric  characterizing  the  "skill  diversity  of  team"  spectrum 

Multi-disciplinary 

Homogeneous 

No  one  particular 
major  holds  a  clear 
higher  concentration 
than  another 

Multiple  majors 
present,  but  a 

concentration  of  one 

particular  major  exists 

Several  members  are 
from  a  different  major 
than  the  majority 

All  but  1-2  members 
are  of  the  same  major 

All  members  are  of  the 
same  major 

Table  6  Rubric  characterizing  the  "scope  of  concern"  spectrum 


Project  team 

Program  office 

Team  is  primarily 
concerned  and 

responsible  for  only  the 
technical  success  of  the 
design 

Team  is  responsible  for 
the  technical  design 
success  within 

copizance  of  basic 

cost  arid  schedule 

constraints 

Team  is  responsible  to 
balance  performance, 

cost  and  schedule  for 
their  design 

Team  is  responsible  for 
most  programmatic 
concerns  in  a  typical 
development  office  as 
part  of  a  company 

Team  is  responsible  for 
alt  programmatic 
aspects  of  running  a 
typical  development 
office  or  company 

1  able  7.  Rubric  characterizing  the  "reflection  of  the  DoD  development  process" 

spectrum 

DoD  Acq 

Rapid/innovation 

Team  is  required  to 
largely  follow  the  DoD 
acquisition  process  and 
related  deliverables 

Team  is  required  to 
follow  some  steps  or 
produce  some 

deliverable  that  are 

defined  in  DoD 

acquisition 

Team  is  aware  of  DoD 
general  acquisition 
process,  but  are 
allowed  to  highly  tailor 
deliverables  without 

adherence  to  DoD 

standards 

Team  follows  a  general 
system  development 
process  without  linkage 
to  DoD  acquisition 

Team  follows  a  novel, 
industry,  or  innovative 
approach  to  system 
development 

Table  8.  Rubric  characterizing  the  "customer  involvement11  spectrum 


Internal/  academic  customer 

External  customer 

Customers  are  the 

Customers  are  largely 

Customers  are  an  equal 

Customer  is  from  an 

Customer  is  from  an 

faculty  administering 

faculty,  but  an  external 

combination  of  faculty 

external  agency  and 

external  agency  and 

the  course  and 

customer  is  involved 

and  external  members, 

drives  the  majority  of 

completely  controls  the 

completely  controls  the 

minimally.  Design 

Design  success  criteria 

the  design  success 

design  success  criteria 

design  success  criteria 

success  criteria  are 

are  developed  in  a 

criteria 

largely  driven  by  the 

balanced  manner 

faculty 

between  the  two  types 

of  customers 

Table  9  Rubric  characterizing  the  "team  size"  spectrum 

Small 

Large 

1  -2  members 

3-5  members 

6-10  members 

1 1-20  members 

21-40  members 
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Table  10.  Rubric  characterizing  the  '  novelty  of  project1'  spectrum 


Original  design 

Modify  an  existing  design 

Project  framework, 
topic,  and  expectations 

are  new  and  not 

previously  offered 

The  project  framework, 
topic  and  expectations 
are  mostly  new,  but 
may  be  slightly  based 
on  previous  research 
offerings 

Ei titer  the  project 
framework  or  topic  has 
been  used  before,  but 

the  other  is  new  for  this 
offering 

The  project  framework, 
topic  and  expectations 
have  been  mostly  used 
before,  but  there  is 

some  small  twist  added 
to  this  offering 

Project  framework, 
topic,  approach,  and 
expectations  already 
exist  Project  has  been 
completed  mostly  in  its 
current  form  before 

Table  11.  Rubric  characterizing  the  "process  formality"  spectrum 

Formal 

Informal 

High  number  of  formal 

and  defined 

deliverables  required 

Several  key  milestone 
deliverables  required 
and  expectation  of 
regular,  prepared  status 

briefs 

Informal  status  briefs, 
but  several  key 
milestone  deliverables 
required 

Minimal  formal 

milestone  deliverables, 
status  briefs  only  as 

needed 

Only  final 
report/briefmg 
expected  as  a 

deliverable 

Table  12.  Rubric  characterizing  the  "potential  for  publishing"  spectrum 

Expected 

Exception 

Publication  in  external, 
peer-reviewed  products 
are  explicitly  expected 
and  regularly  occurs 

Publication  of  results  is 
suggested  and  expected 
for  a  majority  of 
projects 

Publication  of  results 
happens  half  of  the 
time 

Publication  of  results 

may  be  expected  and 
happens  only 
occasionally 

Final  repons  and 
briefings  to  faculty  arc 
sufficient.  Publishing 
results  is  not  suggested 

or  expected  &  happens 
by  very  rare  exception 


Table  13.  Rubric  for  characterizing  the  "system  level" 

spectrum 

Product 

System  of  systems 

A  con sumer- level 
product  being 
developed  as  a  stand¬ 
alone  item 

A  family  of  related 
products  or  a  single 
product  being 
developed  with  its 
associated  logistics, 

maintenance,  and  other 

areas  well-considered 

A  major  subsystem  of  a 
highly  complex  system 
is  being  developed  Or, 
a  small,  but  complete 
system- level  solution  is 
developed  that 
considers  its  full 

context,  interfaces  and 
relation  to  other  system 
issues 

A  system-level  solution 
being  developed 
completely  to  address 
organ  i  zat  i  on  a  1  -  level 
problems  with 
cognizance  of  related 
systems 

A  suite  or  family  of 
systems  being 
developed  and 
integrated  to  address  a 
full,  complex  national 
or  global-level  problem 

Table  14.  Rubric  characterizing  the  "course  maturity" 

spectrum 

New  construct 

Mature  offering 

Project,  instructor,  and 

Two  of  the  following 

Only  one  of  the 

Project,  inst motor,  and 

Project,  instructor,  and 

course  structure  are 

are  new:  project. 

following  are  new: 

course  structure  are 

course  are  established 

brand  new 

instructor,  or  course 

project,  instructor,  or 

established  and  have 

and  have  been  offered 

structure 

course  structure 

been  offered  at  least 

many  times  before 

once  before 

Table  15.  Rubric  of  characterizing  the  "knowledge  use 

:H  spectrum 

New  knowledge 

Applying  know  ledge 
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Team  is  essentially 
viewed  as  completely 

unlearned  to  the 

problem  and  domain 

Large  amounts  of 

instruction  and  research 

needed  to  understand 

major  parts  of  the 
problem  and  domain 

Additional 

instruction/research 

required  to  understand 
a  major  component, 
function,  or  customer 
requirement  for  the 
project  to  be  a  success 

Additional  research 

required  to  understand 
some  subsystem  or 
function  of  the  project 

Only  small  amounts  of 
new  knowledge  are 
required  Project  is 
mostly  application  of 
previously  acquired 
knowledge  to  a 
problem  well-within 

one’s  domain  or 

anpl  icat  ion  d  isc  i  p  1  i  ne 

Table  16.  Rubric  characterizing  the  ’’other  faculty  involvement’1  spectrum 

Single-instructor 

T c  am  of  in  strue  tors 

A  single  instructor 
leads  the  project 
largely  autonomously 

A  single  instructor 
leads  the  project  with 
help/input  from  one  of 
two  others 

A  pair  of  instructors 
co-lead  the  project 

A  few  instructors  are 
used  with  project  lead 
and  instruction  sharing 

Several  instructors  are 
used  to  lead  the  project 
with  input  and 
coverage  from  many 

Table  17.  Rubric  characterizing  the  ’’team  selection’1  spectrum 

Volunteer 

Directed 

Team  is  fully  staffed  by 
vol  urUeer  students  to  a 

specific  project 

Team  is  staffed  mostly 

with  volunteers  to  the 
specific  project 

Team  is  staffed  with 

students  who  have 
chosen  a  project 
domain,  but  not  the 
specific  project 
assigned 

Team  is  staffed  mostly 
with  non-voluneers 

Team  is  fully  staffed  by 

non- volunteer  students 

to  the  domain  and 

project 

Table  18  Rubric  characterizing  the  "competitive"  spectrum 

Sole-Source 

Competition 

Project  has  no  other 
known  competitors 

Project  has  theoretical 
competition  in  the  open 

market  for  the 

developed  solution,  but 

there  is  no  reflection  of 
that  competition  in  the 

course 

Project  has  competition 
through  understanding 
and  analysis  of  the 
open  market  for  their 

solution 

Project  has  similar, 
competing  teams 

within  USAFA 

Project  has  similar, 
competing  teams  at 
externa!  organizations 

Table  19.  Rubric  characterizing  the  "mission  linkage ' 

spectrum 

Military-need 

General  application 

Problem  statement 

uniquely  aligned  with 
operational  military  use 

Problem  is  within  the 
broad,  support  domain 
of  the  military 

Solution  to  problem  is 
considered  a  dual-use 
technology  problem 

Problem  is  defined  in 

general  societal  need 
terms,  however, 
military  application  is 
possible 

Problem  statement  is 

applicable  to  general 
society 

5*  Discussion 

The  absence  of  a  broad,  multi-dimensional  approach  to  characterizing  capstone  experiences  is  currently  missing. 
Before  assessments  can  be  made  about  many,  varied  capstone  projects,  a  framework  for  describing  and 
understanding  them  is  required.  With  the  rubrics  established  in  section  4.2  the  next  phases  of  research  can  begin.  As 
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described  in  section  j.2>  capstone  faculty  will  now  be  asked  to  narrow  the  characteristic  spectrums  of  interest  and 
then  self-evaluate  their  capstone  projects  according  to  the  chosen  spectrum s. 

lt  ls  the  §oal  of  this  research  to  inform  others  in  a  common- language,  effective  method  of  evaluating  different 
projects  against  common  outcomes.  With  the  foundation  of  this  paper  and  subsequent  research  effort  in  this  vein, 
much  can  be  shared  across  institutions  that  promote  interdisciplinary  capstone  experiences.  Much  of  the  framework 
described  can  be  easily  applied  to  other  schools  and  even  within  companies  that  must  assess  relative  performance  of 
multiple  project  divisions. 
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